{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Vectorizing tabular fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook covers simple methods to vectorize tabular data using the same dataset as in the other examples, but this time ignoring the text of the question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "import sys\n",
    "sys.path.append(\"..\")\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "from ml_editor.data_processing import get_normalized_series\n",
    "\n",
    "data_path = Path(\"../data/writers.csv\")\n",
    "df = pd.read_csv(data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's pretend we wanted to predict the **score** from the tags, number of comments, and question creation date. Here is what the data looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Tags</th>\n",
       "      <th>CommentCount</th>\n",
       "      <th>CreationDate</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;resources&gt;&lt;first-time-author&gt;</td>\n",
       "      <td>7</td>\n",
       "      <td>2010-11-18T20:40:32.857</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;fiction&gt;&lt;grammatical-person&gt;&lt;third-person&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-11-18T20:42:31.513</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;publishing&gt;&lt;novel&gt;&lt;agent&gt;</td>\n",
       "      <td>1</td>\n",
       "      <td>2010-11-18T20:43:28.903</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;plot&gt;&lt;short-story&gt;&lt;planning&gt;&lt;brainstorming&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-11-18T20:43:59.693</td>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;fiction&gt;&lt;genre&gt;&lt;categories&gt;</td>\n",
       "      <td>1</td>\n",
       "      <td>2010-11-18T20:45:44.067</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           Tags  CommentCount  \\\n",
       "0                <resources><first-time-author>             7   \n",
       "1   <fiction><grammatical-person><third-person>             0   \n",
       "2                    <publishing><novel><agent>             1   \n",
       "3  <plot><short-story><planning><brainstorming>             0   \n",
       "4                  <fiction><genre><categories>             1   \n",
       "\n",
       "              CreationDate  Score  \n",
       "0  2010-11-18T20:40:32.857     32  \n",
       "1  2010-11-18T20:42:31.513     20  \n",
       "2  2010-11-18T20:43:28.903     34  \n",
       "3  2010-11-18T20:43:59.693     28  \n",
       "4  2010-11-18T20:45:44.067     21  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"is_question\"] = df[\"PostTypeId\"] == 1\n",
    "\n",
    "tabular_df = df[df[\"is_question\"]][[\"Tags\", \"CommentCount\", \"CreationDate\", \"Score\"]]\n",
    "tabular_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to use this data as input to a model, we need to give it a suitable numerical representation. To do so, we will do three things here:\n",
    "\n",
    "1. Normalize numerical input features to limit the impact of outliers\n",
    "\n",
    "2. Transform the date feature in a way that makes it easier to understand for a model.\n",
    "\n",
    "3. Get dummy variables from categorical features so a model can ingest them.\n",
    "\n",
    "First, we normalize the data to reduce the effect of outliers on downstream model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tabular_df[\"NormComment\"]= get_normalized_series(tabular_df, \"CommentCount\")\n",
    "tabular_df[\"NormScore\"]= get_normalized_series(tabular_df, \"Score\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Tags</th>\n",
       "      <th>CommentCount</th>\n",
       "      <th>CreationDate</th>\n",
       "      <th>Score</th>\n",
       "      <th>NormComment</th>\n",
       "      <th>NormScore</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;resources&gt;&lt;first-time-author&gt;</td>\n",
       "      <td>7</td>\n",
       "      <td>2010-11-18T20:40:32.857</td>\n",
       "      <td>32</td>\n",
       "      <td>1.405553</td>\n",
       "      <td>3.66092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;fiction&gt;&lt;grammatical-person&gt;&lt;third-person&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-11-18T20:42:31.513</td>\n",
       "      <td>20</td>\n",
       "      <td>-0.878113</td>\n",
       "      <td>2.02388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;publishing&gt;&lt;novel&gt;&lt;agent&gt;</td>\n",
       "      <td>1</td>\n",
       "      <td>2010-11-18T20:43:28.903</td>\n",
       "      <td>34</td>\n",
       "      <td>-0.551875</td>\n",
       "      <td>3.93376</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;plot&gt;&lt;short-story&gt;&lt;planning&gt;&lt;brainstorming&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-11-18T20:43:59.693</td>\n",
       "      <td>28</td>\n",
       "      <td>-0.878113</td>\n",
       "      <td>3.11524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;fiction&gt;&lt;genre&gt;&lt;categories&gt;</td>\n",
       "      <td>1</td>\n",
       "      <td>2010-11-18T20:45:44.067</td>\n",
       "      <td>21</td>\n",
       "      <td>-0.551875</td>\n",
       "      <td>2.16030</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           Tags  CommentCount  \\\n",
       "0                <resources><first-time-author>             7   \n",
       "1   <fiction><grammatical-person><third-person>             0   \n",
       "2                    <publishing><novel><agent>             1   \n",
       "3  <plot><short-story><planning><brainstorming>             0   \n",
       "4                  <fiction><genre><categories>             1   \n",
       "\n",
       "              CreationDate  Score  NormComment  NormScore  \n",
       "0  2010-11-18T20:40:32.857     32     1.405553    3.66092  \n",
       "1  2010-11-18T20:42:31.513     20    -0.878113    2.02388  \n",
       "2  2010-11-18T20:43:28.903     34    -0.551875    3.93376  \n",
       "3  2010-11-18T20:43:59.693     28    -0.878113    3.11524  \n",
       "4  2010-11-18T20:45:44.067     21    -0.551875    2.16030  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tabular_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's represent dates in a way that would make it easier for a model to extract patterns (see chapter 4 of the attached book for more information on why we chose these particular features.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert our date to a pandas datetime\n",
    "tabular_df[\"date\"] = pd.to_datetime(tabular_df[\"CreationDate\"])\n",
    "\n",
    "# Extract meaningful features from the datetime object\n",
    "tabular_df[\"year\"] = tabular_df[\"date\"].dt.year\n",
    "tabular_df[\"month\"] = tabular_df[\"date\"].dt.month\n",
    "tabular_df[\"day\"] = tabular_df[\"date\"].dt.day\n",
    "tabular_df[\"hour\"] = tabular_df[\"date\"].dt.hour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Tags</th>\n",
       "      <th>CommentCount</th>\n",
       "      <th>CreationDate</th>\n",
       "      <th>Score</th>\n",
       "      <th>NormComment</th>\n",
       "      <th>NormScore</th>\n",
       "      <th>date</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>hour</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;resources&gt;&lt;first-time-author&gt;</td>\n",
       "      <td>7</td>\n",
       "      <td>2010-11-18T20:40:32.857</td>\n",
       "      <td>32</td>\n",
       "      <td>1.405553</td>\n",
       "      <td>3.66092</td>\n",
       "      <td>2010-11-18 20:40:32.857</td>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;fiction&gt;&lt;grammatical-person&gt;&lt;third-person&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-11-18T20:42:31.513</td>\n",
       "      <td>20</td>\n",
       "      <td>-0.878113</td>\n",
       "      <td>2.02388</td>\n",
       "      <td>2010-11-18 20:42:31.513</td>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;publishing&gt;&lt;novel&gt;&lt;agent&gt;</td>\n",
       "      <td>1</td>\n",
       "      <td>2010-11-18T20:43:28.903</td>\n",
       "      <td>34</td>\n",
       "      <td>-0.551875</td>\n",
       "      <td>3.93376</td>\n",
       "      <td>2010-11-18 20:43:28.903</td>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;plot&gt;&lt;short-story&gt;&lt;planning&gt;&lt;brainstorming&gt;</td>\n",
       "      <td>0</td>\n",
       "      <td>2010-11-18T20:43:59.693</td>\n",
       "      <td>28</td>\n",
       "      <td>-0.878113</td>\n",
       "      <td>3.11524</td>\n",
       "      <td>2010-11-18 20:43:59.693</td>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt;fiction&gt;&lt;genre&gt;&lt;categories&gt;</td>\n",
       "      <td>1</td>\n",
       "      <td>2010-11-18T20:45:44.067</td>\n",
       "      <td>21</td>\n",
       "      <td>-0.551875</td>\n",
       "      <td>2.16030</td>\n",
       "      <td>2010-11-18 20:45:44.067</td>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           Tags  CommentCount  \\\n",
       "0                <resources><first-time-author>             7   \n",
       "1   <fiction><grammatical-person><third-person>             0   \n",
       "2                    <publishing><novel><agent>             1   \n",
       "3  <plot><short-story><planning><brainstorming>             0   \n",
       "4                  <fiction><genre><categories>             1   \n",
       "\n",
       "              CreationDate  Score  NormComment  NormScore  \\\n",
       "0  2010-11-18T20:40:32.857     32     1.405553    3.66092   \n",
       "1  2010-11-18T20:42:31.513     20    -0.878113    2.02388   \n",
       "2  2010-11-18T20:43:28.903     34    -0.551875    3.93376   \n",
       "3  2010-11-18T20:43:59.693     28    -0.878113    3.11524   \n",
       "4  2010-11-18T20:45:44.067     21    -0.551875    2.16030   \n",
       "\n",
       "                     date  year  month  day  hour  \n",
       "0 2010-11-18 20:40:32.857  2010     11   18    20  \n",
       "1 2010-11-18 20:42:31.513  2010     11   18    20  \n",
       "2 2010-11-18 20:43:28.903  2010     11   18    20  \n",
       "3 2010-11-18 20:43:59.693  2010     11   18    20  \n",
       "4 2010-11-18 20:45:44.067  2010     11   18    20  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tabular_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally let's transform tags into dummy variables using pandas' [get_dummies](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html) function, with each tag being assigned an index that will take the value \"1\" only if it is present in the given row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select our tags, represented as strings, and transform them into arrays of tags\n",
    "tags = tabular_df[\"Tags\"]\n",
    "clean_tags = tags.str.split(\"><\").apply(\n",
    "    lambda x: [a.strip(\"<\").strip(\">\") for a in x])\n",
    "\n",
    "# Use pandas' get_dummies to get dummy values \n",
    "# select only tags that appear over 500 times\n",
    "tag_columns = pd.get_dummies(clean_tags.apply(pd.Series).stack()).sum(level=0)\n",
    "all_tags = tag_columns.astype(bool).sum(axis=0).sort_values(ascending=False)\n",
    "top_tags = all_tags[all_tags > 500]\n",
    "top_tag_columns = tag_columns[top_tags.index]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>creative-writing</th>\n",
       "      <th>fiction</th>\n",
       "      <th>style</th>\n",
       "      <th>characters</th>\n",
       "      <th>technique</th>\n",
       "      <th>novel</th>\n",
       "      <th>publishing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   creative-writing  fiction  style  characters  technique  novel  publishing\n",
       "0                 0        0      0           0          0      0           0\n",
       "1                 0        1      0           0          0      0           0\n",
       "2                 0        0      0           0          0      1           1\n",
       "3                 0        0      0           0          0      0           0\n",
       "4                 0        1      0           0          0      0           0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_tag_columns.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add our tags back into our initial DataFrame\n",
    "final = pd.concat([tabular_df, top_tag_columns], axis=1)\n",
    "\n",
    "# Keeping only the vectorized features\n",
    "col_to_keep = [\"year\", \"month\", \"day\", \"hour\", \"NormComment\",\n",
    "               \"NormScore\"] + list(top_tags.index)\n",
    "final_features = final[col_to_keep]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>hour</th>\n",
       "      <th>NormComment</th>\n",
       "      <th>NormScore</th>\n",
       "      <th>creative-writing</th>\n",
       "      <th>fiction</th>\n",
       "      <th>style</th>\n",
       "      <th>characters</th>\n",
       "      <th>technique</th>\n",
       "      <th>novel</th>\n",
       "      <th>publishing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "      <td>1.405553</td>\n",
       "      <td>3.66092</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "      <td>-0.878113</td>\n",
       "      <td>2.02388</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "      <td>-0.551875</td>\n",
       "      <td>3.93376</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "      <td>-0.878113</td>\n",
       "      <td>3.11524</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2010</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "      <td>-0.551875</td>\n",
       "      <td>2.16030</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   year  month  day  hour  NormComment  NormScore  creative-writing  fiction  \\\n",
       "0  2010     11   18    20     1.405553    3.66092                 0        0   \n",
       "1  2010     11   18    20    -0.878113    2.02388                 0        1   \n",
       "2  2010     11   18    20    -0.551875    3.93376                 0        0   \n",
       "3  2010     11   18    20    -0.878113    3.11524                 0        0   \n",
       "4  2010     11   18    20    -0.551875    2.16030                 0        1   \n",
       "\n",
       "   style  characters  technique  novel  publishing  \n",
       "0      0           0          0      0           0  \n",
       "1      0           0          0      0           0  \n",
       "2      0           0          0      1           1  \n",
       "3      0           0          0      0           0  \n",
       "4      0           0          0      0           0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_features.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voila! Our tabular data is now ready to be used for a model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml_editor",
   "language": "python",
   "name": "ml_editor"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
